The widespread use of barcodes has greatly simplified supermarket checkout. However, many problems persist, causing both inconvenience for shoppers, and added costs for retailers.
One of the difficulties is finding a barcode on a package. While experienced supermarket clerks eventually learn barcode locations for popular products, even the best clerks sometimes have difficulty with less common products. For shoppers who use self-service checkout stations, any product can be confounding.
Another issue concerns re-orienting a package so that its barcode is in position for reading. Many items are straightforward. However, particularly with large items (e.g., a carton of diapers, or a heavy bag of dog food), it can be a physical challenge to manipulate the product so that the barcode is exposed to the reading device. Often in self-service checkout stations, the physical constraints of the checkout station compound the difficulty, as these stations commonly don't have the handheld scanning capability with which conventional checkouts are equipped—forcing the shopper to manipulate the product so that barcode faces a glass scanning platen on the counter. (When properly positioned, the shopper may be unable to view either the platen or the barcode—exacerbating the difficulty.) Moreover, it is not enough for the barcode to be visible to the scanner; it must also be presented so as to roughly face the scanner (i.e., its surface normal must generally be within about 40-50 degrees of facing the scanning device in order to be read).
Sometimes a product is flipped and turned in search of a barcode, only to find there is none. Bottles of wine, for example, commonly lack barcodes.
Yet another issue is occasional difficulty in getting the scanning equipment to successfully read the barcode, after the barcode has been found and correctly positioned. This is a particular problem with malleable items (e.g., a package of frozen peas), in which the barcoded surface is crinkled or otherwise physically irregular.
To redress such issues, some have proposed identifying products with passive tags that can be sensed by radio (e.g., RFID and NFC chips). However, the costs of these tags are an obstacle in the low-margin grocery business. And it can be difficult to distinguish the responses from several different items on a checkout counter. Moreover, certain materials in the check-out queue may be radio-opaque—preventing some identifiers from being read. Privacy issues raise yet further concerns.
Other checkout technologies have also been tried. For example, in patent publication 20040081799, Kodak describes how a marking can be applied to supermarket packaging by adding a polymer layer that defines scannable information in the form of matte and glossy areas. The matte/glossy areas can form indicia such as barcodes, or digital watermarks. However, this technology requires applying a polymer layer to the packaging—a further expense, and an additional processing step that packagers are not equipped to provide.
Other identification technologies have been proposed for use in conjunction with barcode-based product identification. For example, patent application 20040199427 proposes capturing 2D imagery of products, and checking their color histograms against histograms associated with products identified by sensed barcode data, to ensure correct product identification. The same publication similarly proposes weighing articles on the conveyor—again checking for consistency with the barcode-indicated product. Publications 20040223663 and 20090060259 teach related arrangements, in which imagery of products is used to check for possibly switched barcodes.
Applicant's U.S. Pat. No. 7,044,395 teaches that a watermark can replace a barcode, such as a UPC symbol or other standard product code, in a retail point of sale application. A reader unit at a checkout counter extracts a product identifier from the watermark, and uses it to look up the product and its price.
U.S. Pat. No. 4,654,872 describes a system employing two video cameras, which captures images of a 3D article, and uses the imagery to recognize the article. U.S. Pat. No. 7,398,927 teaches another two-camera system, this one to read product codes from articles despite specular reflections. U.S. Pat. No. 7,909,248 details a self-service checkout terminal in which captured imagery is compared against a database of reference imagery to try to identify a matching product.
In accordance with various embodiments of the present technology, certain drawbacks of the prior art are overcome, and new capabilities are provided.
For example, in one aspect, the present technology involves marking product packaging with a digital watermark that encodes related information (e.g., Universal Product Codes, such as UPC-A or UPC-E; Electronic Product Codes—EPC, European Article Number Codes—EAN, a URI or web address, etc.). The marking spans a substantial part of the packaging surface area, so that it can be sensed from one or more fixed cameras at a checkout station without repositioning of the item. The watermark indicia is applied to the packaging along with other printing—integrated in the other packaging artwork.
In one such embodiment, a variety of recognition technologies are used at a checkout station—looking for different indicia of product identification (watermark, barcode, color histogram, weight, temperature, etc.). The system applies a set of rules to the collected evidence, and outputs a product identification based on the available information.
In another aspect, crinkles and other deformations in malleable product packaging are optically sensed, and are used in decoding an identifier from the distorted surface (e.g., the crinkled surface can be virtually flattened prior to decoding the identifier). In one particular arrangement, the crinkled configuration is sensed by structure-from-motion techniques. In another, the product configuration is sensed by a structured light scanner (e.g., of the sort popularized by the Microsoft Kinect sensor).
In yet another aspect, a checkout station comprises a conveyor belt that includes markings that are optically sensed, and which are used to increase check-out speed and accuracy.
In still another aspect, imagery captured from an item that is being conveyor-transported at a checkout station is processed to compensate for motion blur, prior to applying a product recognition technology.
In yet another aspect, a plenoptic camera system senses information at a checkout station. The collected light field data is then processed to yield multiple different planes of focused imagery, to which product recognition technologies are applied. In some embodiments, these planes include a variety of non-parallel planes.
In still another aspect, 2D imagery that is acquired at a checkout station is applied to a GPU, which computes multiple perspective-transformed versions of the imagery. These different versions of the imagery are then analyzed for product recognition purposes. The GPU can process input imagery of several different focal lengths, e.g., captured by plural fixed-focus cameras, or by a camera that cyclically changes its focal plane, or by plenoptic sensing.
In yet another aspect, piled items presented for checkout are volumetrically modeled and segmented to identify component items in the pile.
In still another aspect, the location of an item that is too obscured to be identified within a pile, is determined, so that a clerk or a mechanical system can expose it for identification.
In yet a further aspect, a confidence score is computed that indicates the certainty of an identification hypothesis about an item. This hypothesis is tested against collected evidence, until the confidence score exceeds a threshold (or until the process concludes with an ambiguous determination).
In still another aspect, data acquired away from the checkout station (e.g., in a store aisle) is used in identifying items at checkout. This data can include, e.g., sensor data evidencing removal of a product from a shelf, location data indicating that the shopper paused near certain merchandise, etc. Such data may be accorded a weight that varies with a time elapsed between its sensing and item checkout.
In yet another aspect, a clerk's or shopper's interaction with an item is sensed to aid in identification of the item. For example, a clerk's gaze may be tracked to identify the location of a salient feature on the item, or a shopper's particular hand pose in grasping the item when putting it into a cart or onto a checkout conveyor may provide some clue about the item's identity.
In still another aspect, a system provides guidance to a clerk or shopper concerning a manner of packing items into bags, e.g., based on the shapes, weights and temperatures of the purchased items.
In yet a further aspect, different items at a checkout station are illuminated with light of different colors, e.g., to indicate items that have been successfully identified (or not), to indicate which items should be placed in which bags, etc.
The foregoing and a great number of other features and advantages of the present technology will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.